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Abstract Body 

Limit 4 pages single-spaced. 

Background / Context: 

Description of prior research and its intellectual context. 

Proficiency-based or competency-based approaches to education are undergoing a 
resurgence in both K-12 and higher education settings, spurred in part by advances in 
digital learning technologies (Priest, Rudenstine, Weisstein, & Gerwin, 2012; Soares, 2012). 
These approaches allow students to progress at their own pace through a diverse range of 
personalized learning experiences aligned to students' interests. In a proficiency-based 
system, students receive credit not as a function of how much time they spend studying a 
subject but, rather, based on demonstrations and assessments of their learning. In addition, 
they are not constrained to progress at the same rate as their peers. The approach allows 
for accelerated learning among students who master the material quickly, and it provides 
additional time and — ideally, additional support — for students who need them. The theory 
is that a larger share of students will ultimately reach proficiency in a given content area if 
they are given the freedom to advance at their own pace and if their learning experiences 
are tailored to their needs and interests (Lewis et al., 2013; Priest, et al., 2012; Sturgis & 
Patrick, 2010). Moreover, policymakers have been responsive to the growing proficiency- 
based education movement. As of March 2012, 36 states had authorized waiver 
mechanisms or other alternative to their seat-time requirements for high school 
graduation, and New Hampshire has maintained a competency-based credit system since 
2009 (Cavanagh, 2012; Grossman & Shipton, 2012; New Hampshire Department of 
Education, 2012). Despite considerable momentum in the field, proficiency-based systems 
are not well-researched. Existing studies have focused largely on narrative descriptions of 
promising models and on interviews with students and practitioners in schools that have 
adapted proficiency-based approaches (Priest, et al., 2012; Sturgis & Patrick, 2010). 

The current study focuses on three distinct approaches to proficiency-based 
education. It attempts to extend the research base by describing the implementation of 
three diverse technology-enabled models and by providing evidence about students’ 
experiences and learning outcomes under each model. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

During the 2011-12 and 2012-13 academic years, the Bill & Melinda Gates 
Foundation funded three organizations in enhancing and refining the use of proficiency- 
based educational models at the high school level. The organizations — which included two 
large school districts and one intermediary organization — carried out the pilot programs in 
a total of 11 high schools distributed across 5 districts. The interventions differed across 
sites and schools but had in common the use of online technology to allow students to 
progress toward mastery along defined pathways, to receive credit for demonstrated 
mastery, and to have access to rich online learning experiences accessible 
anytime/anywhere. The interventions also shared a focus on improved data management 
through development of online systems that would help teachers award credit for mastery. 

Our evaluation of this set of pilot interventions set out to address three core 
questions about the interventions: 
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1) To what extent did the proficiency-based interventions comport with the design 
principles of proficiency-based pathways as described in the grant program 
(pathways to mastery, credit for mastery, and anytime/anywhere learning)? 

2) Among students exposed to the pilot interventions, how did their self-reported 
learning experiences vary within and across programs? 

3) To what extent did students’ exposure to proficiency-based pathways models 
predict their academic performance in the targeted content area of mathematics 
or reading, as measured by performance on state accountability tests? 

Setting: 

Description of the research location. 

The study examines proficiency-based pathways implementation in five settings, each of 
which is summarized, alongside descriptions of the interventions and populations, in Table 1 (see 
Appendix B). Setting A is a suburban district of roughlyl0,500 students, about 72% of which are 
Hispanic or Latino, 19% of which are white, 5% are Asian, 2% are African American or 
American Indian, and a majority of which are eligible for free or reduced-price lunch. Setting B 
is an intermediary organization working with three magnet schools in two large urban districts 
and a proficiency-oriented high school in one small town community. The schools’ percentage of 
minority students in Site B ranges from 5% to 93%, with free/reduced lunch eligibility ranging 
from 35% to 65%. Site C is a large urban district serving more than 140,000 students, of which 
about 56% are African American, 18% are Hispanic or Latino, 14% are white, 7% are Asian, and 
less than one percent are American Indian. A majority of students in Site C are eligible for free 
or reduced-price lunch. 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

As shown in Table 1, the students exposed to the pilot intervention in Site A include 
about 651 students studying mathematics at the eighth or ninth grade level, just under a quarter 
of which are in one middle school, and the remainder of which are in one high school. The 
students are concentrated in targeted classrooms that chose to take part in the pilot. Their 
demographic attributes reflect the district as a whole. Pilot students in Site B include about 2,369 
students in grades 9 through 12, disbursed among four high schools, with demographics as 
described above. The pilot interventions in these sites generally applied to all of the social 
studies and ELA classes in the participating schools. Pilot students in Site C include about 528 
ninth grade students studying English language arts; their demographics reflect the district as a 
whole. They are disbursed among 6 high schools and concentrated within the particular 
classrooms of 7 teachers who chose to participate in the pilot. 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

The interventions, which vary by site, are summarized in Table 1. In Site A, the 
intervention focuses mainly on the development and roll-out of four online mathematics games 
and of teacher-developed instructional videos that they can use to “flip” their classrooms so that 
students can watch lectures outside of class and received personalized instruction during class. In 
Site B, the intervention focuses on portfolio-based demonstrations of proficiency and rubric- 
based grading for clear and frequent feedback. In Site C, the intervention is a set of five 
technology-enhanced, project-and-rubric-based ninth-grade language arts units that encourage 
proficiency-based teaching and learning. Moreover, in all sites, the intervention includes build- 
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out and refinement of a digitally supportive and interoperable learning management system that 
allows teachers and parents to track students’ learning progressions. However, these improved 
learning management systems were not rolled out to all sites during the pilot years. 

Research Design: 

Description of the research design. 

To obtain a quasi-experimental estimate of the pilot interventions’ effects on student 
performance in the targeted subject areas, we adopt a distinct approach for each site to reflect the 
differing scope of implementation across sites. In all cases, we use what amounts to an 
interrupted time-series design with a matched comparison group, but the level of analysis differs 
across sites, as does the construction of the comparison group. In Site A, we are concerned that 
the four mathematics games and flipped classroom videos that were developed as a central part 
of the intervention have too low of a dosage to yield a detectable effect on students’ performance 
in mathematics. Instead, we believe that the most useful analysis for the field of proficiency- 
based learning is one that estimates the holistic effect of the site’s proficiency-based reform on 
student achievement across the district. In 2008-09, the district adopted a “competency-based” 
model of teaching and learning for the entire district, so we propose to compare pre/post grade- 
level trends on the state accountability test between the site and a matched sample of comparison 
schools in the state. We will focus primarily on mathematics to reflect the content-area emphasis 
of the pilot work, however. 

In Site B, the portfolio and rubric-based interventions are carried out largely at the school 
level, so we will draw matched within-district comparisons for each school, with the exception of 
the school in the one-high-school district, for which we will draw matches from a statewide pool. 

In Site C, because the intervention occurs at the classroom level within comprehensive 
high schools, we will use a student-level analysis that compares students in pilot classrooms to 
observably similar students in non-pilot classrooms in the same school, subject (English 
language arts), grade, and year. Additional analytic details are described in the next section. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

To address research question 1, our data collection has included site visits to the program 
schools at the end of pilot Year 1 (June 2012) and during the fall and spring terms of Year 2 
(November 2012 and May 2013, respectively). Data collection at each visit included interviews 
with central administrative staff and pilot teachers, as well as observations of pilot classrooms. 

To address research question 2, in May 2013, we collected a survey data from all the pilot 
school students about their implementation of the intervention, and we will soon examine 
variation in students’ reported experiences at the classroom, teacher, school, and project-site 
levels. These data will allow us to characterize variation in each site’s programmatic approach 
and to examine the extent to which programmatic design corresponds to student self-reports. 

To address research question 3, we use a distinct approach for each site, as described in 
the previous section. In Site A, we are collecting public, school-level performance data in 
mathematics from districts across the state and using propensity scores to match each grade-by- 
year average in the treatment schools to a set of schools matched on student achievement and 
demographics during the period before district-wide competency-based reform took effect in 
2008-09 (Abadie, Drukker, Herr, & Imbens, 2004; Rosenbaum & Rubin, 1983; Rubin, 1997). In 
Site B, we are adopting a similar approach, but as noted above, we are limiting the comparisons 
to within-district schools for the 3 schools located in a large urban district, and using a within- 
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state comparison pool for the school that is the only high school in its district. We will undertake 
the analyses at the grade-by-school-by-year level, though not all high school grades are subject 
to accountability testing, thereby constraining the sensitivity of the analysis to changes in non- 
tested grades. 

For two of the Site B schools, the comparisons will be based on demographic and 
achievement data from the years before the schools began working with the intermediary 
organization that is driving the proficiency-based reform. For the other two Site B schools, which 
are new schools, the comparisons will be based on demo graphically similar schools but will not 
be able to incorporate pre-intervention performance data. Our analysis will use robust standard 
errors that reflect the nesting of year-by-grade observations within schools. 

A limitation of the aggregated-data approach proposed in Sites A and B is that changes in 
student composition may confound the treatment effect estimates, but we will examine the 
magnitude of such changes to assess the extent of likely bias. 

In Site C, the comparison will be undertaken at the student level, using students from the 
same schools, grades, and years who were not exposed to the treatment materials. One 
unavoidable limitation of this approach is that in many cases, the teachers of the comparison 
group will differ from the treatment-classroom teachers, limiting our ability to differentiate 
teacher and curriculum effects. Also, because the intervention was delivered in ninth grade, 
which is not part of the state accountability testing system, our analysis there will not able to 
examine state accountability test scores. The analysis will adjust for demographic attributes and 
for at least one baseline score using student-level benchmark assessment data from the district. 

Findings / Results: 

Description of the main findings with specific details. 

Regarding research question 1, which focuses on design and implementation principles, 
we find that the programs have approached proficiency-based education quite differently, likely 
as a function of their contexts, but that a common thread is the necessity of a strong data 
management system that allows teachers to track student proficiency over time. 

Regarding research questions 2 and 3, student survey and outcome data are currently 
being collected. By September of 2013, we will be able to report in Site A on the estimated 
effects of a district competency-based reform on students’ aggregate performance relative to 
students in a similar set of districts in the same state, matched as a function of demographics and 
prior test score performance. In Site B, we will be able to report on school-level trends relative to 
a matched set of comparison schools from the same district or (in one case) the same state. The 
Site A and B analyses will reflect implementation only through Year 1 of the pilot (2011-12) and 
not yet through the second and final year (2012-13). In Site C, we will be able to compare 
student-level performance on benchmark tests and student attendance at the end of pilot Year 2. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

This study highlights the role of instructional technology in facilitating proficiency-based 
educational approaches — both the ways in which technology is transformative and the ways in 
which it may be useful but less essential. It also lays bare some of the challenges in bringing 
proficiency-based models to scale. Though our analyses of students’ outcomes are still under 
way, the quasi-experimental analytic results will provide new evidence on the near-term effects 
of proficiency-based approaches and pave the way for studies designed to yield more-robust 
causal inferences. 
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Appendix B. Tables and Figures 
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Table 1 . Summary of Sites, Interventions, Populations, and Analytic Comparison Groups 


Program 

Setting 

Pilot scope 

Share of 
minority 
students 

Targeted 
grades and 
content 
areas 

Key intervention 
attributes 

Comparison data 

A 

Suburban 

school 

district 

1 high school 
and 1 middle 
school; about 
651 students 
(10,500 
students in 
district-level 
reform) 

81% 

Grades 8-9 
mathematics 

Online math 
games and 
"flipped" 
classroom 
instruction in the 
context of a 
competency- 
based system 
district wide 

Matched school- 
by-grade-level 
test score 
performance 
within state and 
year 

B 

Intermediary 

organization 

4 high 
schools in 3 
districts; 
about 2369 
students 

71% 

Grade 9-12 
social studies 
& language 
arts 

Portfolio 

assessments with 
rubric-based 
evaluation 

Matched school- 
by-grade test 
score performance 
within district or 
state, and year 

C 

Large, urban 
district 

7 high 
schools; 
about 528 
students 

86% 

Grade 9 
language arts 

Project-based 
learning units 
with rubric-based 
evaluation 

Matched student- 
level benchmark 
data and 

attendance within 
school, grade, and 
year 


SREE Fall 2013 Conference Abstract Template 


B-l 


